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Webinar  Instructions 


-  Software  Engineering  Institute 


Engineering  Improvement  in  Software  Assurance: 
•  M  IT  A  Landscape  Framework 

Carnegie  Mellon  May2oio  4 

©2010  Carnegie  Mellon  University 


Polling  Question  1 


How  did  you  hear  about  this  webinar? 

a)  Email  invitation  from  the  SEI 

b)  SEI  website 

c)  Website  with  webinar  calendar  (i.e.,  www.webinar-directorv.com) 

d)  Social  media  site  (e.g.,  Linkedln,  Twitter) 

e)  Other 
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Agenda 


Problem  Space 

Introduction  to  the  Assurance  Modeling  Framework 
Summary  and  Questions 
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Why  is  modeling  important? 


Modeling  facilitates  understanding  complexity 

•  Mechanisms  to  structure,  describe,  analyze,  and  discuss  complexity 

•  Provides  a  way  to  describe  the  range  of  behaviors  of  the  stakeholders 
involved 

•  Provides  a  way  to  describe  key  social  and  technical  elements  that  must  work 
together  to  achieve  results — a  collaboration  among  solutions  and  participants 

Modeling  to  understand  software  assurance 

•  Numerous  assurance  solutions  (i.e.,  technologies,  policies,  and  practices)  are 
available 

•  A  large  number  of  organizations  produce  or  fund  these  assurance  solutions 

•  Unclear  how  available  assurance  solutions  contribute  to  resulting  operational 
assurance 

•  Need  for  a  way  to  describe  differences  between  available  solutions  and 
assurance  results  (and  how  to  bridge  the  gaps) 
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Assurance  is  More  than  Requirements  Validation 


Software  assurance 

•  Justified  confidence  that  software  functions  as  intended  and  is  free  of 
exploitable  vulnerabilities,  either  intentionally  or  unintentionally  designed  or 
inserted  at  any  time  during  the  life  of  the  software 

Software  context 

•  Functions  as  intended:  includes  user  expectation 

-  Which  will  change  over  time 

•  Context  of  use:  actual  operational  mission  and  environment  of  use 

-  Which  may  or  may  not  be  reflected  in  a  requirements  artifact 
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Multiple  Models  Needed 


Question 

Method  Used  to  Generate 
Models 

1 .  How  is  software  assurance  value  defined  for  a  selected  context? 

Critical  Context  Analysis 

2.  Who/what  are  the  participating  organizations  and  assurance  solutions? 

Value  Mapping 

3.  What  are  the  elements  of  value  exchanged  among  participating  organizations 
and  assurance  solutions? 

Value  Mapping 

4.  How  do  participating  organizations  and  assurance  solutions  work  together  to 
achieve  operational  assurance? 

SoS  Focus  Analysis 

5.  What  are  the  drivers  and  motivations  of  participating  organizations? 

Driver  Identification  and 
Analysis 

6.  What  are  the  critical  usage  scenarios  and  behaviors  among  the  participating 
organizations  and  assurance  solutions? 

System  Dynamics 

7.  What  are  the  adoption  and  operational  usage  mechanisms  used  for  assurance 
solutions?  How  are  they  aligned  with  organizational  contexts  and  needs? 

Technology  Development 
and  Transition  Analysis 

8.  What  is  the  impact  of  future  trends  and  events  on  participating  organizations 
and  assurance  solutions? 

Strategic  Alternatives 
Analysis 

9.  What  patterns  of  possible  inefficiencies  affecting  the  formation,  adoption,  and 
usage  of  assurance  solutions  can  be  identified? 

[informal  analysis] 

1 0.  What  are  candidates  for  improvements?  What  could  be  the  impact,  if 
implemented? 

[informal  analysis] 
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A  Pilot  Using  Vulnerability  Management 


Characteristics  of  the  example 

•  Operational  environments  across  all  domains  are  plagued  with  undiscovered 
defects  and  escalating  numbers  of  known  vulnerabilities 

•  Management  of  vulnerabilities  includes  detection,  remediation,  and 
prevention  activities 

•  Success  requires  the  effective  interactions  of  technologies,  practices,  people, 
and  organizations 

Rich  set  of  available  solutions,  e.g., 

•  Common  Vulnerabilities  and  Exposures  (CVE)@ 

•  Common  Weakness  Enumeration  (CWE)™ 

•  NIST  National  Vulnerability  Database  (NVD) 

•  Static  Analysis  (various  vendor  products) 

•  Secure  coding  practices  (emerging  standards  and  research) 

®  CVE  is  a  registered  trademark  of  The  MITRE  Corporation. 

™  CWE  is  a  trademark  of  The  MITRE  Corporation. 
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Polling  Question  2 


Are  you  familiar  with  vulnerability  management? 

a)  Very  familiar 

b)  Somewhat  familiar  with  the  terms 

c)  No  familiarity 
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Critical  Context  Analysis: 

Principal  Perspectives  &  Influences  (qi,2) 


The  ‘how’: 

How  do  suppliers  organize 
and  constrain  their 
capabilities? 


governance/ 

identity 


how  it  is 
realized 


The  ‘what’: 

What  do  suppliers  do? 


For  a  specific  domain  of  interest 


v  Yellow: 

Brown: 

how  we 
must  do 
what 
we  do 

the  contexts 
from  which 
the  demands 
emerge 

Green: 
what  we  do 

Red: 

particular  — 
demands 

supply  side 


demand  side 


\ 


The  ‘why’: 

What  is  going  on  in  the  larger 
ecosystem  that  makes  what 
suppliers  do  of  value? 


The  ‘for  whom’: 

Who  are  suppliers  serving? 
What  is  the  nature  of  their 
clients’  work? 


Permission  to  use  PAN  technology  in  Critical  Context  Analysis  is  under  license  from  Boxer  Research  Ltd. 
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Critical  Context  Analysis  for  CVE 


Reveals  a  broad  range  of  types  of 
organizations  with  interrelated  roles 


Domain:  CVE  Support  for  Software  Vulnerability  Management 


New  vulnerabilities  registered  in  CVE  list. 
Vulnerability  pattern  determined.  Vulnerability 
data  added  to  NVD. 

CVE  board  monitors  that  new  vulnerabilities 
registered  in  timely  fashion. 

NIST  monitors  use  of  NVD. 

Operational  organizations  of  U.S.  DoD  and 

government  agencies  that  rely  on  computers, 
networks,  software  applications,  data  storage 
media  to  perform  their  mission;  cannot  afford 
loss  of  data  integrity,  data  confidentiality,  and 
availability  for  operations. 

governance/ 

identity 

How  do  suppliers  organize  and 
constrain  their  capabilities? 

What  is  going  on  in  the  larger  ecosystem  that 
makes  what  suppliers  do  of  value? 

h 

how  it  is 

What  do  suppliers  do? 

■ 

Who  are  suppliers  serving?  What  is  the 
nature  of  their  clients’  work? 

realized 

SW  application  vendors:  build,  test,  issue 
patches  for  vulnerabilities.  Register  patches  in 
CVE  list. 

SW  security  product  vendors:  build,  test, 
issue  a  capability  to  detect/contain  a 
vulnerability.  Cross  reference  to  CVE  ID. 

supply  side:  managing  vulnerabilities 

Site  security  analysts:  track  vulnerabilities  and 
available  patches;  form  site  specific  solutions;  and 
notify  IT  ops  of  vulnerabilities  and  solutions. 

IT  operations:  track  and  install  available  site 
solutions;  get  computer  users  to  install  patches, 
and  monitor  for  compliance. 

demand  side:  concerned  with  assurance  of 
operational  systems 
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Legend 


Symbols 


participant  J  a  participant  (e  g.,  organization  or  technology)  in  a  value  exchange 
Data  source  for  public  information  with  multiple  contributors 


Line  Style 


- ►  Dashed  arrow  Value  is  pulled  by  destination  organization. 

- ►  Solid  arrow  Value  is  pushed  from  source  organization. 

Note :  The  direction  of  the  arrow  shows  the  flow  of  the  value  exchange. 

Line  Colors 


Green 

Funding 

Blue 

Product 

■>  Brown 

Service 

>  Gray 

Governance 

*  Red 

Compliance 

Orange 

Endorsement 

Software  Engineering  Institute 


Value  Mapping: 

Value  Exchanged  (02,3, 4) 


Partial  CVE  Diagram 
Notation  Example 


Microsoft,  Apple. 
Symantec,  others 


Reporting 

Product 

Vendors 

(Trusted*) 

/ 

/ 

1 

Solution 

Information 
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Supplying  CVE 


Public  source 
vulnerability  data*' 


CVE  Diagram 

as  of  31  March  2009 


Microsoft.  Apple. 
Symantec,  others 


r^j,,  Independent 
organizations 
collaborate  with 
minimal  formalities 


We  are  working  with 
networks  or  lattices 
of  relationships 


DoD.  OMB, 
DHS.  GSA. 
CIO  Council 
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CVE  for  IT 
Operations 


ph 


“Distance”  between 
an  assurance 
solution  and 
operational  use  is 
often  large  and 
complex 


Includes  site 
security  analysts 


IT  Operations  (DoD) 


Products  focused 
on  viruses,  static 
analysis,  security 


Site  policies 
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Resources  Roles 


SoS  Focus  Analysis: 

Potential  Assurance  Results  <02, 4) 


Supply  Side  (provided  capabilities) 


Layers 


what  do  we  have 
to  do 

how  do  we  need  to 
organize  these 
activities 

who  are  our  customer/users 
for  this  work 

why  -  what  is 
driving  the 
need  for  this 
demand 

^ ^  ^ ^ 

^ ^ 

;  ^ ^ 

Technology 
elements 
(HW,  SW) 

Technical 
integration  of 
elements 

- ^ 

Generalized 

operational 

capabilities 

Orchestration 
of  capabilities 
in  an 

operational 

environment 

- ^ 

Operational 
performance 
of  the 
capability 

^ ^ 

Operational  i 
outcome 
achieved  for 
particular 
context  of  use 

Demand  Side  (actual  operational  uses) 


Permission  to  use  PAN  technology  in  SoS  Focus  Analysis  is  under  license  from  Boxer  Research  Ltd. 
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Resources  Responsibilities  Roles 


SoS  Focus  Analysis 
for  CVE 


Strong  emphasis  on  supply-side  assurance  solutions. 

Areas  of  potential  inefficiencies:  where  tacit  knowledge 
is  held  and  people  manually  synthesize  significant 
information  from  multiple  sources. 


What 

Vendors 

How 

CVE,  NVD 

Who 

Security 

analysts 

Who 

Computer 
installations  & 
operations 

Why 

User 

environments 

1 

I 

1 

1 

1 

Addressing 

1 

Disseminating 

Maintaining 

i  i 

Maintaining 

i  \ 

Maintaining 

/  \ 

Operational 

known 

vulnerabilities 

current 

current 

awareness  of 

assurance  in 

vulnerabilities 

and  patches 

knowledge  of 
vulnerabilities 
and  patches 

knowledge  of 
available 
patches  &  site 
configurations; 
forming  site 
solutions 

risks  and 
effectiveness  of 
solutions 

the  context  of 

use 

Building, 

testing, 

issuing 

patches 

Registering 

Supply  Side 

Monitoring 

Tracking, 

analyzing, 

forming 

solutions 

Installing 

solutions, 

monitoring 

effectiveness 

Demand  Side 

Operational 
availability  and 
integrity 

Layers  l 


2 


3 


4 


5 


6 
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Polling  Question  3 


How  would  you  characterize  the  focus  of  your  organization? 

a)  Supply  Side 

b)  Demand  Side 
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System  Dynamics:  Critical  Behaviors  (Q6> 


3.  The  proactive  approach  focuses  on  a  strategy  of 
vulnerability  prevention  based  on  applying  CWE 
information  within  the  vendor  community  to  developed 
software  that  prevents  vulnerabilities. 


Vendor  Community 
vuls  in  newly 
developed  softwari 


Vendor  Community 
correcting  vul 
prevention  problems 


Proactive  Product 
Vulnerability 
Prevention 


disseminating 
CWE  software 
weaknesses 


Vendor  Community 
vul  prevention  training 
and  experimentation 

^  .+ 


Vendor 
Community 
resources  to 
vul  prevention’ 


1 .  Vendors  must  decide  how  to  split  resources  between 
reactive  and  proactive  responses  to  product 
vulnerabilities  to  balance  the  need  for  an  immediate 
response  with  the  need  for  a  proactive  solution  that 
prevents  product  vulnerabilities. 


Vendor  Community 
product  vuls 


Vendor  Community 
patching  product  vuls 


Vendor  Resource 
Reallocation 


Reactive  Product 
Vulnerability 
Patching 


Vendor 
Community 
resources 
to  patch  , 


disseminating 
CVE  software 
vuls 


Vendor 
Communit 
urgency  o: 
response 


4.  If  vendors  feel  the  need  to  devote  more 
resources  to  vulnerability  patching  and  less  to 
vulnerability  prevention,  then  this  leads  to  a 
downward  spiral  of  increasingly  vulnerable 
products  and  ever  increasing  assurance  problems. 


2.  The  reactive  approach  patches  product 
vulnerabilities  based  on  CVE  information.  The 
development  of  patches  is  prioritized  based,  in 
part,  on  the  impact  a  given  vulnerability  is  having 
on  the  operational  community. 
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Detailed  System  Dynamics  Model 


=  Software  Engineering  Institute 
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Transition  Analysis:  Adoption  of  Products  (Q7) 

Issue — maturation  and  transition  models  built  for  single  technologies 
and  not  clusters  of  technologies 


Subprocesses:  Building  the  Value  of  a  New  Technology 

I 


I 


I 


I 


Bridges:  Satisfying  and  Mobilizing  Stakeholders  at  Each  Stage 


Source:  V.  Jolly,  Commercializing  New  Technologies:  Getting  from  Mind  to  Market,  1997. 
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Extracted  Success  Indicators 


What  does  success  mean  for 
assurance  solutions?  Market 
share?  Improved  operational 
assurance  of  some  %  of 
operational  organizations? 


Indicators  of  Maturation  and  Adoption  Success  for  CVE 

CVE  is  accepted  throughout  the  supplier  community. 

CVE  is  considered  a  de-facto  standard  by  the  community. 
Vendors  advertise  that  they  are  CVE  compliant. 

Content  providers/list  makers  reference  vulnerabilities  using  CVE. 
NVD  explicitly  uses  CVE. 


Factors  Contributing  to  Success  for  CVE 

MITRE  identified  a  clear  market  need  (from  a  community  perspective). 

Vendors  were  motivated  to  participate. 

MITRE’s  strategy  allowed  it  to  partner  with  researchers  and  content  providers/list  makers. 

Agrowing  amount  of  vulnerability  information  was  distributed  across  multiple  databases  (operated  by  competing  groups). 

MITRE  filled  an  unmet  community  need  with  CVE. 

MITRE  signed  agreements  with  vendors  to  get  information  earlier. 

MITRE’s  proof  of  concept  using  public  data  convinced  vendors  of  the  value  of  the  CVE  approach. 

MITRE  identified  the  right  stakeholders  and  did  a  good  job  of  getting  them  involved  in  building  the  solution 

MITRE  explicitly  focused  on  reducing  the  barriers  to  adoption 

MITRE’s  solution  did  not  force  adopters  to  change  the  way  they  did  business. 

Government  policy  -  DoD  IAVA  was  rewritten  to  include  CVE. 

MITRE  continues  CVE  “marketing”  and  product  evolution. 

There  is  continued  investment  in  infrastructure. 

Community  articulated  “standard”  before  MITRE  used  the  term. 

Focus  on  building  collaborations. 
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Transition  Analysis  Insights 


Technology  maturation  and  transition  mechanisms  for  CVE  are  being 
applied  to  CWE 

•  CVE  required  little  behavioral  change  on  the  part  of  its  primary  users  (e.g., 
suppliers  of  IT  and  vulnerability  management  products) 

•  CWE  will  require  extensive  behavioral  and  process  changes  on  the  part  of  its 
primary  users  (e.g.,  software  development  organizations) 

There  are  other  critical  differences  among  the  user  communities 

•  CVE:  characterizes  vulnerabilities  from  an  operational  perspective — written  in 
the  language  of  operations 

•  CWE:  characterizes  weaknesses  associated  with  vulnerabilities  from  a 
software  development  perspective — written  in  the  language  of  software 
engineering 
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Applying  the  Assurance  Modeling  Framework 


includes  decision  makers, 
technologies,  practices, 
people,  and  their  relationships 


describes  landscape  of 
assurance  ecosystem  for 
selected  assurance 
capability  area  to  better 
inform  resource  decisions 


facilitates  creation  of  a  profile  of  selected 
assurance  capability  area  based  on  important 
aspects/elements  of  assurance  ecosystem 


select  assurance 
capability  area  for  an 
assurance  property 


select  assurance 
solutions  that  claim  to 
provide  the  assurance 
capability 


Assurance 

Modeling 

Frameworl 
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Value  of  this  Work 


Modeling  addresses  key  questions 

•  Where  are  the  critical  gaps  in  available  assurance  solutions? 

•  Where  should  resources  be  invested  to  gain  the  most  benefit? 

•  What  additional  assurance  solutions  are  needed? 

•  Are  the  incentives  for  routinely  applying  assurance  solutions  effective? 

Assurance  modeling  framework  lays  important  groundwork  by  providing 
a  multi-dimensional  approach  to 

•  Understanding  relationships  between  organizations  and  assurance 
solutions — how  these  relationships  contribute  to  operational  assurance 

•  Identifying  potential  areas  for  improvement  across  a  spectrum  of  technical 
and  organizational  areas 
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Polling  Question  4 


Would  this  modeling  approach  be  useful  to  your  organization? 

a)  Very  useful 

b)  Somewhat  useful 
o)  Not  at  all 
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Current  Work 


Detailed  report  of  framework  and  its  pilot  application  to  vulnerability 
management  under  final  review  (available  summer  2010) 

Apply  the  framework  to  a  second  assurance  capability  area 

•  Selected  malicious  software  prevention  and  management 

•  Expand  understanding  of  the  customer/user  (i.e.,  the  demand  side) 

Conducted  interviews  and  constructed  initial  models  from  the  demand 
side 

•  Information  Security  Office 

•  IT  operations 

•  CSIRT 
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NO  WARRANTY 


THIS  CARNEGIE  MELLON  UNIVERSITY  AND  SOFTWARE  ENGINEERING  INSTITUTE 
MATERIAL  IS  FURNISHED  ON  AN  “AS-IS"  BASIS.  CARNEGIE  MELLON  UNIVERSITY 
MAKES  NO  WARRANTIES  OF  ANY  KIND,  EITHER  EXPRESSED  OR  IMPLIED,  AS  TO 
ANY  MATTER  INCLUDING,  BUT  NOT  LIMITED  TO,  WARRANTY  OF  FITNESS  FOR 
PURPOSE  OR  MERCHANTABILITY,  EXCLUSIVITY,  OR  RESULTS  OBTAINED  FROM 
USE  OF  THE  MATERIAL.  CARNEGIE  MELLON  UNIVERSITY  DOES  NOT  MAKE  ANY 
WARRANTY  OF  ANY  KIND  WITH  RESPECT  TO  FREEDOM  FROM  PATENT, 
TRADEMARK,  OR  COPYRIGHT  INFRINGEMENT. 

Use  of  any  trademarks  in  this  presentation  is  not  intended  in  any  way  to  infringe  on  the  rights 
of  the  trademark  holder. 

This  Presentation  may  be  reproduced  in  its  entirety,  without  modification,  and  freely 
distributed  in  written  or  electronic  form  without  requesting  formal  permission.  Permission  is 
required  for  any  other  use.  Requests  for  permission  should  be  directed  to  the  Software 
Engineering  Institute  at  permission@sei. cmu.edu. 

This  work  was  created  in  the  performance  of  Federal  Government  Contract  Number  FA8721  - 
05-C-0003  with  Carnegie  Mellon  University  for  the  operation  of  the  Software  Engineering 
Institute,  a  federally  funded  research  and  development  center.  The  Government  of  the  United 
States  has  a  royalty-free  government-purpose  license  to  use,  duplicate,  or  disclose  the  work, 
in  whole  or  in  part  and  in  any  manner,  ana  to  have  or  permit  others  to  do  so,  for  government 
purposes  pursuant  to  the  copyright  license  under  the  clause  at  252.227-7013. 
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Questions? 


Contact  Information 


Lisa  Brownsword 

Senior  Member,  Technical  Staff 
Research,  Technology,  and 
System  Solutions  (RTSS) 
Program 

+1  703-908-8203 
llb@sei.cmu.edu 

Christopher  J.  Alberts 

Senior  Member,  Technical  Staff 
Acquisition  Support  Program 
(ASP) 

+1  412-268-3045 
cia@sei.cmu.edu 


Carol  C.  Woody,  PhD. 

Senior  Member,  Technical  Staff 
Networked  Systems  Survivability 
(NSS)  Program 
+1  412-268-9137 

cwoodv@cert.org 


Andrew  P.  Moore 

Senior  Member,  Technical  Staff 
Networked  Systems  Survivability 
(NSS)  Program 
+1  412-268-5465 

aom@cert.org 
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